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Abstract 

An exact solution of the transient dynamics for a sequential associative memory model is dis- 
cussed through both the path-integral method and the statistical neurodynamics. Although the 
path-integral method has the ability to give an exact solution of the transient dynamics, only 
stationary properties have been discussed for the sequential associative memory. We have suc- 
ceeded in deriving an exact macroscopic description of the transient dynamics by analyzing the 
correlation of crosstalk noise. Surprisingly, the order parameter equations of this exact solution 
are completely equivalent to those of the statistical neurodynamics, which is an approximation 
theory that assumes crosstalk noise to obey the Gaussian distribution. In order to examine our 
theoretical findings, we numerically obtain cumulants of the crosstalk noise. We verify that the 
third- and fourth-order cumulants are equal to zero, and that the crosstalk noise is normally dis- 
tributed even in the non-retrieval case. We show that the results obtained by our theory agree 
with those obtained by computer simulations. We have also found that the macroscopic unstable 
state completely coincides with the separatrix. 
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I. INTRODUCTION 



Statistical mechanical theories have been applied to the field of neural networks such 
as combinatorial optimization and associative memories. Recently, they have also been 
applied to the field of information science such as error- correcting codes, image recovery, and 
CDMA [0, 0, ^ |], U, ||, 0, |J. For example, the maximum a posteriori probability (MAP) 
and the maximum posterior marginals (MPM) in the framework of the Bayesian estimation 
correspond to finding the ground state and the equilibrium state of the corresponding spin 
system, respectively. From this point of view, information processing can be treated as 
some kind of relaxation process of spin systems. The transient dynamics of systems should 
therefore be discussed from the information theoretic point of view. 

In general, however, theoretical treatment of the dynamics is extremely difficult compared 
with the equilibrium statistical mechanics of frustrated systems into which many problems 
concerning the information processing are mapped. Although a lot of works has been done 
to analyze the transient dynamics || |K], |TTJ, it is well known that achieving the tractable 
rigorous treatment is hopeless regarding the dynamics of frustrated systems. In this paper, 
we discuss a correlation type associative memory model having frustrated couplings, and 
rigorously derive an exact macroscopic description of the transient dynamics for this model. 

There are two types of associative memory models: autoassociative memory models and 



sequential associative memory models. In the case of autoassociative memory models |T2 



their storage capacity and phase diagram have been analyzed by equilibrium theories Ul3|, [H 



In the case of sequential associative memory models, there is no equilibrium state, since the 
network retrieves different patterns sequentially. As we mentioned above, since associative 
memory models definitely belong to information processing systems, it is important to ana- 
lyze the information process, that is, the retrieval process of the stored patterns. Although 



the path-integral method based on the generating function || [15], [16|, [17j has the potential 
ability to provide a rigorous solution for an associative memory model, the theory is formal 
and intractable because the complexity of the numerical calculation is exponentially large 
with respect to the time step. Accordingly, the theory can only describe the short time 
region and equilibrium state. 

Recently, During et al. (18| presented the path-integral method regarding a sequen- 
tial associative memory model. However, they did not discuss the transient properties of 
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the retrieval process, but only analyzed properties of the stationary state, i.e., the storage 
capacity and the phase diagram [Tj|. On the other hand, the statistical neurodynamics 



I9| , 2D| , p2[ is an approximation theory capable of analyzing the long-term behavior 
of the transient dynamics. Here, the input to the spin or neuron can be divided into two 
terms. The first one is a signal term, which is a signal to retrieve, and the second one is a 
crosstalk noise term, which prevents retrieval. In the statistical neurodynamics, one assumes 
that the crosstalk noise obeys the Gaussian distribution with mean and a time dependent 
variance, and derives macroscopic recursive equations for the amplitude of the signal term 
and the variance of the crosstalk noise. Therefore, the basin of attraction and other dynamic 
properties can be discussed through the statistical neurodynamics. For the autoassociative 
memory model, Nishimori and Ozeki tried to numerically check the Gaussian assumption 
using large-scale computer simulations, and found that the assumption holds only when the 
retrieval process succeeds [^(| p3fl . 



In this paper, we derive an exact solution of the transient dynamics for the sequential 
associative memory model through the path-integral method, and derive an approximated 
solution through the statistical neurodynamics. We then compare the results of the two 
theories, and try to clarify their relationship. 

This paper is constructed as follows. In the second section, the sequential associative 
memory model that we use is defined. In the third section, we introduce the formal macro- 
scopic state equations obtained by During et al. using the path-integral method. In the 
fourth section, we derive an exact macroscopic description of the transient dynamics by the 
path-integral method. In the fifth section, we also derive macroscopic state equations by the 
statistical neurodynamics, and prove that both the path-integral method and the statistical 
neurodynamics give the same equations for the sequential associative memory model. In the 
sixth section, the transient dynamics are verified by comparing these theories with computer 
simulations. 

II. SEQUENCE PROCESSING NEURAL NETWORK 

Let us consider a sequential associative memory model that consists of iV spins or neurons. 
We consider the case of N — > oo. The state of the spins takes (Ji(t) = ±1 and updates the 
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state synchronously with the following probability: 

Probst + 1)1/^)] = -[l + <7;(t + l)tanh (1) 

N 

iH{t) = E J «°-i(*) + ^(*). ( 2 ) 

i=l 

where /3 is the inverse temperature, /3 = 1/T. When the temperature is absolute zero, i.e., 
T = 0, the state of the spins cr^ (t + 1) is determined by the sign of local field hi(t), that is, 

t7 i (t + l) = sgn[^(t)]. (3) 

The term 9i(t) is a threshold or an external input to the network. Synaptic connection 
stores p random patterns £ M = • • • ,£tv) t so as to retrieve the patterns as — > £ 2 — > 
■ ■ ■ £ p — > sequentially. For instance, it is given by 

where $, p+1 = The number of stored patterns p is given by p = aN, where a is called 
the loading rate. Each component of the patterns is assumed to be an independent random 
variable that takes a value of either +1 or —1 according to the following probability, 

Prob[tf = ±l] = i (5) 

We determine the initial state <r(0) according to the following probability distribution, 

ProbMO) = ±1] = I±^<°£, (6) 

and therefore the overlap between the pattern £ p and the initial state <r(0) is m(0). The 
network state cr{t) at time t is expected to be near the pattern £*, when initial overlap m(0) 
is large and the loading rate is small. 



III. PATH-INTEGRAL METHOD 



During et al. [18|] discussed the sequential associative memory model by the path-integral 
method. In this section, we introduce macroscopic state equations for the model with a finite 
temperature T > 0, according to their paper. The detailed derivation is available in their 
paper |18 



In order to analyze the transient dynamics, the generating function Z\^>\ is defined as 
m = E pKOl.^llr-^Wle- 1 ^^, (7) 

<T(0),-,<7(t) 



where V = ("0(0), • • • , i/>(t — 1)). The state tr(s) = (ai(s), ■ ■ ■ , &n(s)) T denotes the state of 
the spins at time s, and the probability p [er(0), <r(l), • ■ ■ , cr(t)] denotes the probability of 
taking the path from initial state <x(0) to state a(t) at time t through cr(l), <x(2), ■ • ■ , cr(t— 1). 
As ([?]) shows, the generating function takes the summation of all 2 N ( t+1 > paths, which the 
network can take from time to t. The generating function Z[i[>] involves the sequence 
overlap m(s), which represents the direction cosine between the state cr(s) and the retrieval 
pattern £ s at time s, the response functions G(s,s'), and the correlation functions C(s,s') 
as follows: 
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(8) 
(9) 
(10) 



where (■) denotes the thermal average. Using the assumption of self- averaging, we replace 
the generating function Z[ij)\ with its ensemble average Z\ip\. 

Evaluating the averaged generating function Z[ip] through the saddle point method, 
During et al. succeeded in obtaining the following macroscopic recursive equations for the 
order parameters of (p|)-(|ITj|). 

w w-\w Rw tanh/? r^ m(s + e ^ s _ ^ + _ ^ 
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(11) 



(12) 



{dvdw} e 

x tanh/3 £ s m(s - 1) + 9(s - 1) + \/av(s - 1) 
x tanh/3 k s 'm(s - 1) + 0(s' - 1) + \/av(s' - 



(13) 



where {dvdw} 



n 



dv(s) dw(s) 



for 



(«(0), «(!),•■■,«(* - 1)) T , w 



(w(0), to(l), • • ■ , w(t — 1)) T , (•)* denotes the average over all £'s, and the matrix R is given 
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by 



r=j:[(gtc(& 



n>0 



(14) 



IV. DYNAMIC TREATMENT OF THE PATH-INTEGRAL METHOD 



A. Transient dynamics 



These formal dynamical equations seem to be intractable because the numerical com- 
plexity in directly calculating these equations becomes exponentially large with respect to 
the time s. Although the rigorous solution of these equations formally has an ability to 
treat the macroscopic state transition, During et al. derived stationary state equations from 
these formal dynamical equations, and only analyzed the stationary properties of the storage 
capacity and the phase diagram. In contrast, we have succeeded in obtaining a tractable 
description of the macroscopic dynamic state transition from the result as shown below. 

Since the matrix R is R(s,s') = (v(s)v(s')), the matrix R represents the covariance 
matrix of crosstalk noise v. We need the value of R(s, s') for each time to analyze the 
transient dynamics exactly. In consideration of this, we reconsidered ([12]) and succeeded 
in deriving a recurrence relation form of R(s,s'). From (|T2"D, G(s, s') = is satisfied when 
s^s' + l. Therefore, the matrix G can be given by 

•■■ 

01 ■■■0 
G = g 2 ••• , (15) 

••• 0_ 

with g s = G(s, s — 1). From this, we can easily find that G n (s, s') = S S)S ' +n YIt=o 9s-t, {n > 1) 
and 

'c(s,s') ,71 = 

n— 1 Ti— 1 

C(s - n, s' - n) J| g s - T J[ gv_ r / , 1 < n < s . (16) 

t=0 r'=0 

,n > s 

From ([14]) , R(s, s') can be reduced to 

s n— 1 n—1 

R(s, s') = C(s, s') + C{s - n, s' - n) J] G{s - r, s - r - 1) J[ G(s' - r', s' - r' - 1) 



'G) n C (G f 



n>l 



r=0 



r'=0 



(17) 



C(s, s') + C(s -l,s'- l)G(s, s - l)G(s', s' - 1) 



s-l 



+ ^C(s-n-M'-n-l)nG(s-r,s-r-l)n G(s' - r', s' - r - 1) (18) 

n>l t=0 t'=0 

f 

C(s, sT) + G(s, s - l)G{s', s' -1)1 C(s - 1, s' - 1) + C{s - n - 1, s' - n - 1) 

[ n>l 

n— 1 n— 1 ^ 

x J] G{jb - t - 1, s - t - 2) J] G{s' - t' - 1, s' - t' - 2) \ (19) 

r=0 t'=0 J 



We can therefore derive a recurrence relation form of R(s, s'): 

R(s, s') = C(s, s') + G(s, s - l)G(s', s' - l)R(s - 1, s' - 1). 



(20) 



Using this recurrence relation, we can evaluate the value of R(s, s') for each time. 

Next, since the terms tanh(-) of m(s) and G(s, s — 1) include only the variable v (s — 1), 
these multiple integral equations can be reduced to the following single integral equations 
by using ( |A18| ) in the Appendix^. 



m(s) = (e s J L>ztanh/3 
G(s, s-l) = (3^1-Q Dztantf 



Z"m(s - 1) + 9(s - 1) + zJaR(s - 1, s - 1 







i s m(s - 1) + 0(s - 1) + zJaR(s - 1, s - 1) 



(21) 



(22) 



with the familiar abbreviation = 2^ 2 . Since C(s, s') includes v(s — 1) and t> (s' — 1) 

C(s, s') can be reduced to a double integral equation from ( |A18|) , 



Cm(s - 1) + 6(s - 1) + zJaR(s - 1, s - 1 



C(s,s) = 1 

C(s,0) = ^ p m(0) J Dztanhfi 
= 

/ r dze ii ^ r 
C(s, s') = { — tanh /3 fm(s - 1) + 0(s - 1) + - 1) 

V 27r|iJn|a L 

x tanh/? k s 'm(s' - 1) + 0(s' - 1) + y/az(s' - 1 



(23) 
(24) 
(25) 



(26) 

where the matrix Rn is a 2 x 2 matrix consisting of the elements of R at time s — 1 and 
time s' — 1, and z = [z(s — 1), z(s' — 1)] T . We therefore have the macroscopic state equations 
(pOj) — (|26|) for each time, and can analyze the transient dynamics exactly. 
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B. Stationary state equations 



Let us derive stationary state equations from our macroscopic state equations. We assume 
m(t) — > m and R(t, t') — > r when t, t' — ► oo. In this case, we can get 



m 



q 



£ / .Dztanh/3 £m + 9 + zy/ar 
Dztanh 2 (3 £m + 9 + z\far 



and also i — 1) — > (5 (1 — q). We can, therefore, obtain 

1 



\2 ' 



i-/3 2 (i-<?y 

These equations (|27|)-(^9D are coincident with those obtained by During el al. Using 
(E9l), the stationary state can be evaluated. 



(27) 
(28) 

(29) 



V. STATISTICAL NEURODYNAMICS 



A. Finite temperature case 

Macroscopic state equations have also been derived for a sequential associative memory 
model with absolute zero temperature T = by the statistical neurodynamics |2~5 |. In 
this paper, we derive the macroscopic state equations for the present model with finite 
temperature T > 0, by the statistical neurodynamics |[20|| . In the statistical neurodynamics, 
the input is divided into a signal term and a crosstalk noise term. From (0) and (Q), we 
obtain 



fn( 8 ) = ^ + m s (s) + z t (s) + 9 t (s), 

Zi(s) = Ee +1 ^( S ), 



(30) 
(31) 

(32) 



The overlaps m M (s) are defined for each pattern £ M . We assume that the crosstalk noise 
Zi(s) is normally distributed with mean and variance p 2 (s, s). Then, the sequence overlap 
m(s) = m s (s) for condensed patterns becomes 



mis 



C J Dztanh(3[Cm(s - 1) + 9(s - 1) + zp(s -1,8- . 



(33) 



Next, let us evaluate the overlap m M (s),/i 7^ s between the network state cr(s) and the 
uncondensed pattern £ M in order to calculate the covariance matrix p 2 (s, s') = E [zi(s)Zi(s')\ 
of the crosstalk noise of ([H]). We assume that the pattern £f and the state cx^s) are 
independent regarding i. The state (Tj(s) is correlated with the uncondensed pattern £f, 
because the local field hi(s — 1) includes the pattern From (POD, since the overlap m M (s) 
is 0(1/ VN), the term tanh [/3/i»(s — 1)], which determines the state Oi(s) stochastically, is 
expanded as follows: 



tanh [/^(s - 1)] = tanh [f3h!f\s - 1)J + pgmT\8 - 1) sech 2 [/to^s - 1) 
fc^a - 1) = ffm- 1 ^ - 1) + i «rV( a - 1) + - 1). 

U^S,fJ,— 1 

Here, we use (tanh a?) = 1 — tanh 2 x = sech 2 x. Then, we get 



(34) 
(35) 



Prob[«r i (s)|/i < (s- 1)] = Prob [ai(s)\h^\s - 1) 

1 



+ 



1 + pg<Ti{8)-mr\a - 1) sech 2 (3h^\s - 1) 



We can derive 



™"00 = ^E^W^K^-i)- 



(36) 



(37) 



Note that o^\s) is independent of £f, and U(s) is given by 

C/( s ) = p(jDz sech 2 /? [( s m(s - 1) + 6(s - 1) + zp(s - 1)]' 

jl - ^ Z^tanh 2 /? [£ s m(s - 1) + 6(s - 1) + zp(s - 1)]^ | . 



(38) 
(39) 



Therefore, the covariance of the crosstalk noise is given by 
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From (37), the third term on the RHS of fl40f) becomes 



(40) 
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1 E Y.^T\sm 



N 
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U(s')E 



1 t E^rM M ^~v-i; 



N 2 



+U{s')U{s' - l)E 



N 



(41) 



According to the literature |24j], (|4l|) becomes zero as follows. Since £f and £f _1 are inde- 
pendent of 0$ (s) and a^ 1 ~ 1 '(s' — 1), respectively, the first term on the RHS of C 4lf) becomes 



CCrM^K-^ ^O' - 1) =0. Using (0) to © up to the initial time iteratively, (gl 



becomes 



AT 



n 



T=l 



(42) 



Because of the sequential associative memory with p = aN period, the correlations for 
s' = p, 2p, 3p ■ ■ ■ remain. These can however be neglected in the limit N — > oo. Therefore, 



the third term on the RHS of ( f4"0"|) becomes zero. Similarly, the forth term also becomes 
zero. Finally, the covariance can be given by 



p\s,s') = aC(s,s') + U(s)U(s')p 2 (s-l,s' - 1). 



(43) 



B. Path-integral method and statistical neurodynamics 

Let us compare the results obtained by the path-integral method and the results obtained 
by the statistical neurodynamics. From (p0|) and fl43|) , let p 2 (s,s') correspond to aR(s,s'). 
In this case, we obtain U(s) = G(s, s — 1) from (^) and (ft^), and then p 2 (s, s 1 ) = aR(s, s 1 ) 
from (|20| ) and (H). Moreover, the overlap from ( |2~1~D is equal to one from (p3|). 

As stated above, both theories give the same macroscopic state equations for the se- 
quential associative memory model. This finding means that the crosstalk noise in the 
present model is normally distributed even if the network fails in retrieval, and also that the 
macroscopic state equations obtained by the statistical neurodynamics can give the exact 
solution. 
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VI. RETRIEVAL PROCESS 



In this section, let us discuss the transient dynamics of the sequential associative memory 
model. Since macroscopic state equations obtained by the statistical neurodynamics are 
equivalent to those by the path-integral method, we use the notations of m(t),U(t), and 
p 2 (t,t'). Let p 2 {t,t) be p 2 {t,t) = ar(t) when t' = t. We analyze the case of 6{t) = and 
derive the macroscopic state equations, 



(44) 



m(t + l) = (^ t+1 J Dz tanh/5 £ m m(t) + zyj ' ar{t) 

U(t + 1) = (3^1 - Q Dz tanh 2 (3 £ w m{t) + z^ocr(t) ^ j, (45) 
r(t + l) = l + U 2 (t + l)r(t). (46) 

Figure 1 shows the time evolution of the overlap m(t). The solid lines represent results 
obtained by the macroscopic state equations (144]) — (f46~l) , and the broken lines represent results 
obtained by computer simulations with iV = 100000, where the loading rate is a = 0.20, 0.26 
and the inverse temperature is j3 = 5 (T = 0.2). The abscissa denotes the time t, and the 
ordinate denotes the overlap m(t). In this case, the storage capacity is a c = 0.246. As 
shown in Fig. 1, our theory can describe the transient dynamics quantitatively even if the 
network fails in retrieval or the loading rate a exceeds the storage capacity a c . 

In the autoassociative memory models, the crosstalk noise is not normally distributed 



when the network fails in retrieval |20], ^3[. In the present sequential associative memory 
model, however, the macroscopic state equations obtained by the path-integral method are 
represented in the form of a single or double Gaussian integral. Therefore, the first, second, 
third, and fourth cumulants Ci(t), C^t), C^t), C±(t) are evaluated in order to verify the 
distribution of the crosstalk noise. When the crosstalk noise is normally distributed, the 
third and fourth cumulants are zero. The cumulants are defined by 



Cx{t) = z{t), 

C 2 (t) = ^(t)-z 2 (t), 

C 3 {t) = - 3?(t)F(t) + 22 s (t), 

C 4 (t) = ¥(t) - 3 (F(t)) 2 - Az(t)^(t) + 12^ 2 (t)F(t) - 6^(t), 



(47) 
(48) 
(49) 
(50) 
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where z n (t) denotes the nth-order moment for the crosstalk noise Zi(t) and is defined as 

^(*)=i!? Etewr- ( 5i ) 

iv i=l 

The cumulants Ci(t) and C 2 (t) represent the average and variance of z(t), respectively. 

Figure 2 shows the time evolution of cumulants and overlap m(t), where the loading 
rate is a = 0.20,0.26 and the inverse temperature is (3 = 5 (T = 0.2). The initial overlap 
is m(0) = 0.2, where the network fails in retrieval. The solid lines represent the overlap 
m(t) and the variance of the crosstalk noise r(t) obtained by our theory. These lines agree 
with the overlap and the cumulant C^if) obtained by the computer simulations. Since 
the third and fourth cumulants are zero, we can find that the crosstalk noise is normally 
distributed. In the sequential associative memory model, the third and fourth cumulants 
hold Cs(t) = Ci(t) = 0, as obtained by the path-integral method, even if the network fails in 
retrieval. The Figure 2 shows that the Gaussian assumption in the statistical neurodynamics 
folds for the present model. 

Figure 3 shows the transition of the overlap m(t) and the variance of the crosstalk noise 
ar(t). The solid lines represent results obtained by our theory and the broken lines represent 
results obtained by computer simulations with N = 100000. The loading rate is a = 0.20 
and the inverse temperature is (3 = 5 (T = 0.2). As shown in Fig. 3, there are two stable 
stationary states; one is m pa 1, which represents a retrieval state, and the other is m = 0, 
which represents a non-retrieval state. We can therefore define the critical overlap, which 
determines whether patterns will be retrieved. When the initial overlap is larger than the 
critical overlap m c , stored patterns can be retrieved. Namely, as shown in Fig. 3, there 
is a separatrix on the locus starting from the initial overlap m c , and the network passes 
through the separatrix at time t = 1. In general, an unstable stationary state obtained 
by macroscopic state equations does not agree with a separatrix obtained by microscopic 
dynamics, e.g., ([!]). As shown by the cross mark in Fig. 3, however, we can find that the 
unstable stationary state agrees with the separatrix for the present model. 

Figure 4 shows a basin of attraction where the inverse temperature is f3 — 5. The abscissa 
denotes the loading rate a, and the ordinate denotes the overlap. The solid line represents 
a result obtained by theory. The lower line represents the critical overlap m c , and the upper 
line represents the stationary overlap at time t — > oo, where the network starts from the 
initial overlap m(0) = 1. The intersection between the critical overlap m c and the stationary 
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overlap gives the storage capacity a c . The broken lines represent the overlap m, which 
is the unstable stationary state given by equations (p7|)-(|29|). The error bars represent the 
median, and the first and third quartiles over 11 trials of the overlaps m c and m^, which 
are obtained by computer simulations with N = 10000. As shown in this figure, the basin 
of attraction obtained by the theory quantitatively coincides with that of the computer 
simulations. 



VII. CONCLUSIONS 

We rigorously derived dynamic macroscopic state equations for the sequential associative 
memory model through the path-integral method. This suggested that the crosstalk noise 
is normally distributed for the sequential associative memory model. We also examined the 
cumulants of the crosstalk noise by computer simulations, and found the third and fourth 
cumulants to be zero. This finding strongly supported the normally distributed crosstalk 
noise. Using this exact solution, we obtained dynamical properties such as the critical overlap 
m c , the temporal behavior of the overlap, and so on. These theoretically obtained results 
agreed with those of computer simulations. We also found that the separatrix coincides with 
the unstable fixed point of the macroscopic stationary state equations. 

Historically, dynamic macroscopic state equations have been derived by the statistical 
neurodynamics for a sequential associative memory model with zero temperature T = 
assuming the crosstalk noise to obey the Gaussian distribution p5| . We extended this 



theory to the finite temperature case, and compared dynamic macroscopic state equations 
by the statistical neurodynamics with those by the path- integral method. As a result, we 
found both to be completely equivalent for the sequential associative memory model. This 
means that the statistical neurodynamics is exact for the sequential associative memory 
model. 

For the sequential associative memory model, we could drive the recurrence relation of 
R(s, s') and the dynamic macroscopic state equations. This is the reason why no effective 
self- interaction, which is caused by the time correlation of states, remains. In contrast, since 
only one pattern is retrieved for an autoassociative memory model, the effect of the intrinsic 
time correlation can not be avoided and the distribution of the crosstalk noise becomes 
non-Gaussian. It is known that this treatment of the autoassociative memory model is 
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intractable |9], 



26 



APPENDIX A: MULTIPLE GAUSSIAN INTEGRAL 



In order to calculate equations ( pi]) to (|i3|), it is sufficient to treat the following form of 
a multiple Gaussian integral, 

{dvdw}e iV - w - 1 > w - Rw f(v 1 ), (Al) 

where we put v = (v\, v 2 ) T . We define S as the inverse matrix of R. Corresponding to the 
representation of v = (v\, V2) T , the matrix R and its inversion S are represented as, 

/ 



R 



Rn R12 
R21 R22 



Sn S12 
S21 S22 



(A2) 



For example, v 1 and Rn are v(s — 1) and R(s — l,s — 1) with equations (pTT|) and (|12|). 
respectively, while = (t>(s — 1), t>(s' — 1)) T and 

Rn J R{a - l ' S - l) ^-^'-M, (A3 ) 
yR(s' -l,s-l) R(s' - l,s' - 1) i 

with equation flTB|). According to the definition, SR = I. We obtain 

Sh.Rh + S 12 R 2 i = I (A4) 

S 2X Rn + S 22 R 2 i = 0, (A5) 

where J and are the identity and zero matrix, respectively. Solving (^4) and (A"5) with 
respect to Ru, we obtain Rn = (S\\ — S^S^S '21 



Accordingly, the integral of equation ( |A"T| ) is obtained as 



{dvdw} e ™-w-iw-Rw f M (A6) 



f , , . , -±(w-iR 1 vV R(w iR 1 v)-±v-R 1 v , . . 
/ {dvdw} e V / \ / / (v^ 



(A7) 



l T ( {dv}e^ vR ^ v f( Vl ) (A8) 



IRI* 



f ( dv \ e -%(v 1 -SuV 1 +V 1 -Si2V a +V a -S2 l Vi+V2.SvV2)f r v \ /J±g\ 
|R|a J 

1 f f 1 1 -|ft'2+5 22 1 5l 2 i;iy522ft'2+5^ 1 52iVi N )-i'ulf5i 1 -5 1 25^ 1 52i N )'Ui , . , 

— - / {dv}e v ; v ; *v y 

R h J 
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H\ 2 1 022 1 2 

for v = (u(0), v(l), ■■■,v(t- 1)) T , and {d«i} 



(AlO) 
(All) 



dv(s) 

" r[ ' . Substituting the following identities, 



with {cfo} = n s <t 
_ _i) < 



dt,( - s - or {dvi} 



SioSno So-i 



<3ll — Ji20 2 2 J 21 

^21 >^22 

5n — S12 S 2 2 S21 1 1 S'22 1 



I5HJII 



into equation ([All]) , we get 

, p|1 , 1 c , /We-W"'/(«i) 

XL 2 |<J22| 2 

= —^-t I {dv 1 }e-^ R ^f(v 1 ). 
\Rn\ 2 J 

We can therefore obtain a simple form of the multiple integral, 



{dvdw}e iVW - 1 z wRw 



(A12) 

(A13) 
(A14) 



(A15) 

(A16) 
(A17) 



\R n \ 2 J 



(A18) 
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FIG. 1: Time evolution of overlap m(t) with loading rate a = 0.20,0.26 and inverse temperature 
(3 = 5. The solid lines denote theoretical results and the broken lines denote results by computer 
simulations with TV = 100000. 
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FIG. 2: Time evolution of cumulants C\(t), C^t), C3 (t) , C4(t) and overlap m(t) with loading rate 
a = 0.20, 0.26 and inverse temperature (3 = 5. The initial overlap is m(0) = 0.2. The solid 
lines denote theoretical results and the broken lines denote results by computer simulations with 
N = 100000. 
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FIG. 3: Overlap m(t) and variance of crosstalk noise r(t) with loading rate a = 0.20 and 
inverse temperature (3 = 5. The solid lines denote theoretical results and the broken lines 
denote results by computer simulations with N = 100000. The initial overlap is m(0) = 
0.10, • • • , 0.40, 0.43, 0.44, 0.50, • • • , 1.0. 
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FIG. 4: Basin of attraction (solid line) obtained by theory with inverse temperature (3 = 5. The 
error bars denote the median, and first and third quartiles over 11 trials by computer simulations 
with N = 10000. 
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